Ab>initio, Big Data, Informatica, Tableau, Data Architect, Cognos, Microstrategy, Healther Business Analysts, Cloud etc.

Job Title : Senior AWS Data Engineer

Experience : 5+ Years

Location : Gurugram

Employment Type : Full-Time

Job Summary :

Seeking a Senior AWS Data Engineer with expertise in AWS to design, build, and optimize scalable data pipelines and data architectures. The ideal candidate will have experience in ETL/ELT, data warehousing, and big data technologies.

Key Responsibilities :

• Build and optimize data pipelines using AWS (Glue, EMR, Redshift, S3, etc.).
• Maintain data lakes & warehouses for analytics.
• Ensure data integrity through quality checks.
• Collaborate with data scientists & engineers to deliver solutions.

Qualifications :

• 7+ Years in Data Engineering.
• Expertise in AWS services, SQL, Python, Spark, Kafka.
• Experience with CI/CD, DevOps practices.
• Strong problem-solving skills.

Preferred Skills :

• Experience with Snowflake, Databricks.
• Knowledge of BI tools (Tableau, Power BI).
• Healthcare/Insurance domain experience is a plus.

Job Description:

The data science team is responsible for solving business problems with complex data. Data complexity could be characterized in terms of volume, dimensionality and multiple touchpoints/sources. We understand the data, ask fundamental-first-principle questions, apply our analytical and machine learning skills to solve the problem in the best way possible. 

Our ideal candidate

The role would be a client facing one, hence good communication skills are a must. 

The candidate should have the ability to communicate complex models and analysis in a clear and precise manner. 

The candidate would be responsible for:

• Comprehending business problems properly - what to predict, how to build DV, what value addition he/she is bringing to the client, etc.
• Understanding and analyzing large, complex, multi-dimensional datasets and build features relevant for business
• Understanding the math behind algorithms and choosing one over another
• Understanding approaches like stacking, ensemble and applying them correctly to increase accuracy

Desired technical requirements

• Proficiency with Python and the ability to write production-ready codes. 
• Experience in pyspark, machine learning and deep learning
• Big data experience, e.g. familiarity with Spark, Hadoop, is highly preferred
• Familiarity with SQL or other databases.

1. Big data developer with 8+ years of professional IT experience with expertise in Hadoop ecosystem components in ingestion, Data modeling, querying, processing, storage, analysis, Data Integration and Implementing enterprise level systems spanning Big Data.
2. A skilled developer with strong problem solving, debugging and analytical capabilities, who actively engages in understanding customer requirements.
3. Expertise in Apache Hadoop ecosystem components like Spark, Hadoop Distributed File Systems(HDFS), HiveMapReduce, Hive, Sqoop, HBase, Zookeeper, YARN, Flume, Pig, Nifi, Scala and Oozie.
4. Hands on experience in creating real - time data streaming solutions using Apache Spark core, Spark SQL & DataFrames, Kafka, Spark streaming and Apache Storm.
5. Excellent knowledge of Hadoop architecture and daemons of Hadoop clusters, which include Name node,Data node, Resource manager, Node Manager and Job history server.
6. Worked on both Cloudera and Horton works in Hadoop Distributions. Experience in managing Hadoop clustersusing Cloudera Manager tool.
7. Well versed in installation, Configuration, Managing of Big Data and underlying infrastructure of Hadoop Cluster.
8. Hands on experience in coding MapReduce/Yarn Programs using Java, Scala and Python for analyzing Big Data.
9. Exposure to Cloudera development environment and management using Cloudera Manager.
10. Extensively worked on Spark using Scala on cluster for computational (analytics), installed it on top of Hadoop performed advanced analytical application by making use of Spark with Hive and SQL/Oracle .
11. Implemented Spark using PYTHON and utilizing Data frames and Spark SQL API for faster processing of data and handled importing data from different data sources into HDFS using Sqoop and performing transformations using Hive, MapReduce and then loading data into HDFS.
12. Used Spark Data Frames API over Cloudera platform to perform analytics on Hive data.
13. Hands on experience in MLlib from Spark which are used for predictive intelligence, customer segmentation and for smooth maintenance in Spark streaming.
14. Experience in using Flume to load log files into HDFS and Oozie for workflow design and scheduling.
15. Experience in optimizing MapReduce jobs to use HDFS efficiently by using various compression mechanisms.
16. Working on creating data pipeline for different events of ingestion, aggregation, and load consumer response data into Hive external tables in HDFS location to serve as feed for tableau dashboards.
17. Hands on experience in using Sqoop to import data into HDFS from RDBMS and vice-versa.
18. In-depth Understanding of Oozie to schedule all Hive/Sqoop/HBase jobs.
19. Hands on expertise in real time analytics with Apache Spark.
20. Experience in converting Hive/SQL queries into RDD transformations using Apache Spark, Scala and Python.
21. Extensive experience in working with different ETL tool environments like SSIS, Informatica and reporting tool environments like SQL Server Reporting Services (SSRS).
22. Experience in Microsoft cloud and setting cluster in Amazon EC2 & S3 including the automation of setting & extending the clusters in AWS Amazon cloud.
23. Extensively worked on Spark using Python on cluster for computational (analytics), installed it on top of Hadoop performed advanced analytical application by making use of Spark with Hive and SQL.
24. Strong experience and knowledge of real time data analytics using Spark Streaming, Kafka and Flume.
25. Knowledge in installation, configuration, supporting and managing Hadoop Clusters using Apache, Cloudera (CDH3, CDH4) distributions and on Amazon web services (AWS).
26. Experienced in writing Ad Hoc queries using Cloudera Impala, also used Impala analytical functions.
27. Experience in creating Data frames using PySpark and performing operation on the Data frames using Python.
28. In depth understanding/knowledge of Hadoop Architecture and various components such as HDFS and MapReduce Programming Paradigm, High Availability and YARN architecture.
29. Establishing multiple connections to different Redshift clusters (Bank Prod, Card Prod, SBBDA Cluster) and provide the access for pulling the information we need for analysis. 
30. Generated various kinds of knowledge reports using Power BI based on Business specification. 
31. Developed interactive Tableau dashboards to provide a clear understanding of industry specific KPIs using quick filters and parameters to handle them more efficiently.
32. Well Experience in projects using JIRA, Testing, Maven and Jenkins build tools.
33. Experienced in designing, built, and deploying and utilizing almost all the AWS stack (Including EC2, S3,), focusing on high-availability, fault tolerance, and auto-scaling.
34. Good experience with use-case development, with Software methodologies like Agile and Waterfall.
35. Working knowledge of Amazon's Elastic Cloud Compute( EC2 ) infrastructure for computational tasks and Simple Storage Service ( S3 ) as Storage mechanism.
36. Good working experience in importing data using Sqoop, SFTP from various sources like RDMS, Teradata, Mainframes, Oracle, Netezza to HDFS and performed transformations on it using Hive, Pig and Spark .
37. Extensive experience in Text Analytics, developing different Statistical Machine Learning solutions to various business problems and generating data visualizations using Python and R.
38. Proficient in NoSQL databases including HBase, Cassandra, MongoDB and its integration with Hadoop cluster.
39. Hands on experience in Hadoop Big data technology working on MapReduce, Pig, Hive as Analysis tool, Sqoop and Flume data import/export tools.

About Telstra

Telstra is Australia’s leading telecommunications and technology company, with operations in more than 20 countries, including In India where we’re building a new Innovation and Capability Centre (ICC) in Bangalore.

We’re growing, fast, and for you that means many exciting opportunities to develop your career at Telstra. Join us on this exciting journey, and together, we’ll reimagine the future.

Why Telstra?

• We're an iconic Australian company with a rich heritage that's been built over 100 years. Telstra is Australia's leading Telecommunications and Technology Company. We've been operating internationally for more than 70 years.
• International presence spanning over 20 countries.
• We are one of the 20 largest telecommunications providers globally
• At Telstra, the work is complex and stimulating, but with that comes a great sense of achievement. We are shaping the tomorrow's modes of communication with our innovation driven teams.

Telstra offers an opportunity to make a difference to lives of millions of people by providing the choice of flexibility in work and a rewarding career that you will be proud of!

About the team

Being part of Networks & IT means you'll be part of a team that focuses on extending our network superiority to enable the continued execution of our digital strategy.

With us, you'll be working with world-leading technology and change the way we do IT to ensure business needs drive priorities, accelerating our digitisation programme.

Focus of the role

Any new engineer who comes into data chapter would be mostly into developing reusable data processing and storage frameworks that can be used across data platform.

About you

To be successful in the role, you'll bring skills and experience in:-

Essential 

• Hands-on experience in Spark Core, Spark SQL, SQL/Hive/Impala, Git/SVN/Any other VCS and Data warehousing
• Skilled in the Hadoop Ecosystem(HDP/Cloudera/MapR/EMR etc)
• Azure data factory/Airflow/control-M/Luigi
• PL/SQL
• Exposure to NOSQL(Hbase/Cassandra/GraphDB(Neo4J)/MongoDB)
• File formats (Parquet/ORC/AVRO/Delta/Hudi etc.)
• Kafka/Kinesis/Eventhub

Highly Desirable

Experience and knowledgeable on the following:

• Spark Streaming
• Cloud exposure (Azure/AWS/GCP)
• Azure data offerings - ADF, ADLS2, Azure Databricks, Azure Synapse, Eventhubs, CosmosDB etc.
• Presto/Athena
• Azure DevOps
• Jenkins/ Bamboo/Any similar build tools
• Power BI
• Prior experience in building or working in team building reusable frameworks,
• Data modelling.
• Data Architecture and design principles. (Delta/Kappa/Lambda architecture)
• Exposure to CI/CD
• Code Quality - Static and Dynamic code scans
• Agile SDLC      

If you've got a passion to innovate, succeed as part of a great team, and looking for the next step in your career, we'd welcome you to apply!

___________________________

We’re committed to building a diverse and inclusive workforce in all its forms. We encourage applicants from diverse gender, cultural and linguistic backgrounds and applicants who may be living with a disability. We also offer flexibility in all our roles, to ensure everyone can participate.

To learn more about how we support our people, including accessibility adjustments we can provide you through the recruitment process, visit tel.st/thrive.

• 4-10 years of experience in software development.
• At least 2 years of relevant work experience on large scale Data applications.
• Strong coding experience in Java is mandatory
• Good aptitude, strong problem solving abilities, and analytical skills, ability to take ownership as appropriate
• Should be able to do coding, debugging, performance tuning and deploying the apps to Prod.
• Should have good working experience on
• o Hadoop ecosystem (HDFS, Hive, Yarn, File formats like Avro/Parquet)
• o Kafka
• o J2EE Frameworks (Spring/Hibernate/REST)
• o Spark Streaming or any other streaming technology.
• Strong coding experience in Java is mandatory
• Ability to work on the sprint stories to completion along with Unit test case coverage.
• Experience working in Agile Methodology
• Excellent communication and coordination skills
• Knowledgeable (and preferred hands on) - UNIX environments, different continuous integration tools.
• Must be able to integrate quickly into the team and work independently towards team goals

• Take the complete responsibility of the sprint stories' execution
• Be accountable for the delivery of the tasks in the defined timelines with good quality.
• Follow the processes for project execution and delivery.
• Follow agile methodology
• Work with the team lead closely and contribute to the smooth delivery of the project.
• Understand/define the architecture and discuss the pros-cons of the same with the team
• Involve in the brainstorming sessions and suggest improvements in the architecture/design.
• Work with other team leads to get the architecture/design reviewed.
• Work with the clients and counter-parts (in US) of the project.
• Keep all the stakeholders updated about the project/task status/risks/issues if there are any.